R tools at Utah DWQ

Jake Vander Laan ()

Use arrow keys or navigator in bottom right to navigate slides. Press escape for presentation overview.


Why use open source, code-based workflows?

  • Transparency
  • Repeatability
  • Efficiency
  • Flexibility
  • Ownership

What do we these tools for?

  • Querying/downloading data
  • Water quality assessments
  • Data visualization
  • Documenting & communicating analyses
  • Wasteload analysis (in development)

Our general approach

  1. Reusable, generalizable functions with flexible inputs
  2. Use arguments to modify functions, instead of re-programming
  3. Link functions in series to perform analyses
  4. Document functions and analyses
  5. Use interactive tools, figures, and applications to enhance efficiency and user understanding

What tools do we use?

R

R is an open source statistical program and coding language.
r-project.org


Robust user community of coders & problem solvers.
Fully customizable.

R

Functions & packages

  • Function:
    A block of code that performs a particular task or set of tasks. A function has a name, arguments, a body (code), and a return value.

  • Argument:
    Change-able inputs to a function.

  • Package:
    A portable collection of R functions.

R documentation

Markdown

Text formatting syntax that allows conversion of plain text to rich text HTML, PDF, and other document types including tables, figures, and code.

  • Documents (package rmarkdown)
  • Chapters & books (package bookdown)
  • Presentations
  • Dashboards
  • Websites

R Markdown - Bookdown

Compile documents into books. Document complex processes & user build guides.

Example: Utah IR R tools user guide

Data portals & web services

Water quality data portal
waterqualitydata.us/portal

Data portals & web services

Web service capabilities

Data portals & web services

two_site_nr=readWQP(type="narrowresult",
                    siteid=c("UTAHDWQ_WQX-4900440","UTAHDWQ_WQX-4900470"), 
                    start_date="01/01/2016", end_date="12/31/2018")

utah_lake_nr=readWQP(type="narrowresult",
                     auid=c('UT-L-16020201-004_01', 'UT-L-16020201-004_02'),
                     start_date="01/01/2016", end_date="12/31/2018",
                     siteType="Lake, Reservoir, Impoundment")

Data portals & web services

ECHO

Data portals & web services

ECHO

UT0021717_tp_001=readECHO_ec(p_id="UT0021717", 
    parameter_code="00665", outfall="001", print=F)
## [1] "https://echodata.epa.gov/echo/eff_rest_services.download_effluent_chart?p_id=UT0021717&parameter_code=00665&outfall=001"
UT0021717_tp_001_effluent=subset(UT0021717_tp_001, 
    monitoring_location_desc=="Effluent Gross")
Permit ID Date Parameter Result Units Stat
UT0021717 01/31/2023 Phosphorus, total [as P] 1.3 mg/L 30DA AVG
UT0021717 12/31/2022 Phosphorus, total [as P] 2.9 mg/L 30DA AVG
UT0021717 11/30/2022 Phosphorus, total [as P] 2.7 mg/L 30DA AVG
UT0021717 10/31/2022 Phosphorus, total [as P] 2.4 mg/L 30DA AVG
UT0021717 09/30/2022 Phosphorus, total [as P] 2.5 mg/L 30DA AVG
UT0021717 08/31/2022 Phosphorus, total [as P] 2.5 mg/L 30DA AVG

GitHub

Collaborative code writing, versioning, & sharing platform.
Individuals, organizations, teams, and repositories.

GitHub

DWQ GitHub organization
A location for DWQ staff to store, collaborate, and share repositories.
github.com/utah-dwq

DWQ R packages
DWQ has two packages in current development. These can be viewed and installed via GitHub.

R-Shiny

Shiny is an R package for building interactive web applications to visualize and analyze data and communicate analytical results.

Examples

Querying & processing data

Full page

Salinity in Great Salt Lake

An analysis of changes in salinity in Great Salt Lake in response to changes to in hydrologic connectivity through a railroad causeway that bisects the lake.

GSL Salinity Dashboard

Utah’s Lake Dashboard

Interactively visualize lake profile & trophic indicator data.
View on the DWQ Shiny apps server Source code

Utah Lake Data Explorer

Analyze and visualize water quality data in Utah Lake.

View on the DWQ Shiny apps server

Source code

TMDL development

Fremont River E.coli TMDL

TMDL development

View on the DWQ Shiny apps server

Source code

TMDL development

Timeseries

TMDL development

Monthly trends

TMDL development

Load duration curves

Integrated Report


Assembling data

Water Quality Portal 

library(wqTools)
downloadWQP(outfile_path='C:/Your/Folder/Path', 
            start_date="01/01/2018", end_date="12/31/2018", 
            retrieve=c("narrowresult","sites","detquantlim","activity"))

Other datasets

  • High frequency DO
  • Macroinvertebrates
  • Phytoplankton

Screening data

  • Table-based validation
  • Tables are unique-d subsets of metadata
  • Reviews accumulate cycle-to-cycle
  • Functions to update & apply data validation tables
  • Results in screened data with consistent nomenclature



Screening data

  1. Labs & activity types
  2. Activity media
  3. Parameter names & fractions
  4. Detection conditions
  5. Detection limit types
  6. Unit conversion factors
  7. Apply screening tables and subset to accepted data

Screening data

Assigning criteria

assignCriteria(data, crit_wb, crit_sheetname, ss_sheetname,
               crit_startRow = 1, ss_startRow = 1, rm_nocrit = TRUE)

Formula derived criteria

Hardness, temperature, & pH dependent criteria

Assessment methods as functions

Methods




Functions

Count exceedances


Assess counts

Review tools

Interactive tools for visualizing sites, data, and assessment results and providing reviewer feedback.


Questions

jakevl.github.io/utah-dwq-rtools-presentation